{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第三章 PyTorch基础：Tensor和Autograd\n",
    "\n",
    "## 3.1 Tensor\n",
    "\n",
    "Tensor，又名张量，读者可能对这个名词似曾相识，因它不仅在PyTorch中出现过，它也是Theano、TensorFlow、\n",
    "Torch和MxNet中重要的数据结构。关于张量的本质不乏深度的剖析，但从工程角度来讲，可简单地认为它就是一个数组，且支持高效的科学计算。它可以是一个数（标量）、一维数组（向量）、二维数组（矩阵）和更高维的数组（高阶数据）。Tensor和Numpy的ndarrays类似，但PyTorch的tensor支持GPU加速。\n",
    "\n",
    "本节将系统讲解tensor的使用，力求面面俱到，但不会涉及每个函数。对于更多函数及其用法，读者可通过在IPython/Notebook中使用函数名加`?`查看帮助文档，或查阅PyTorch官方文档[^1]。\n",
    "\n",
    "[^1]: http://docs.pytorch.org"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.1.0a0+65d6f10_2_ged1fa68'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's begin\n",
    "from __future__ import print_function\n",
    "import torch  as t\n",
    "t.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  3.1.1 基础操作\n",
    "\n",
    "学习过Numpy的读者会对本节内容感到非常熟悉，因tensor的接口有意设计成与Numpy类似，以方便用户使用。但不熟悉Numpy也没关系，本节内容并不要求先掌握Numpy。\n",
    "\n",
    "从接口的角度来讲，对tensor的操作可分为两类：\n",
    "\n",
    "1. `torch.function`，如`torch.save`等。\n",
    "2. 另一类是`tensor.function`，如`tensor.view`等。\n",
    "\n",
    "为方便使用，对tensor的大部分操作同时支持这两类接口，在本书中不做具体区分，如`torch.sum (torch.sum(a, b))`与`tensor.sum (a.sum(b))`功能等价。\n",
    "\n",
    "而从存储的角度来讲，对tensor的操作又可分为两类：\n",
    "\n",
    "1. 不会修改自身的数据，如 `a.add(b)`， 加法的结果会返回一个新的tensor。\n",
    "2. 会修改自身的数据，如 `a.add_(b)`， 加法的结果仍存储在a中，a被修改了。\n",
    "\n",
    "函数名以`_`结尾的都是inplace方式, 即会修改调用者自己的数据，在实际应用中需加以区分。\n",
    "\n",
    "#### 创建Tensor\n",
    "\n",
    "在PyTorch中新建tensor的方法有很多，具体如表3-1所示。\n",
    "\n",
    "表3-1: 常见新建tensor的方法\n",
    "\n",
    "|函数|功能|\n",
    "|:---:|:---:|\n",
    "|Tensor(\\*sizes)|基础构造函数|\n",
    "|tensor(data,)|类似np.array的构造函数|\n",
    "|ones(\\*sizes)|全1Tensor|\n",
    "|zeros(\\*sizes)|全0Tensor|\n",
    "|eye(\\*sizes)|对角线为1，其他为0|\n",
    "|arange(s,e,step|从s到e，步长为step|\n",
    "|linspace(s,e,steps)|从s到e，均匀切分成steps份|\n",
    "|rand/randn(\\*sizes)|均匀/标准分布|\n",
    "|normal(mean,std)/uniform(from,to)|正态分布/均匀分布|\n",
    "|randperm(m)|随机排列|\n",
    "\n",
    "这些创建方法都可以在创建的时候指定数据类型dtype和存放device(cpu/gpu).\n",
    "\n",
    "\n",
    "其中使用`Tensor`函数新建tensor是最复杂多变的方式，它既可以接收一个list，并根据list的数据新建tensor，也能根据指定的形状新建tensor，还能传入其他的tensor，下面举几个例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-8.5769e-19,  4.5622e-41,  0.0000e+00],\n",
       "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 指定tensor的形状\n",
    "a = t.Tensor(2, 3)\n",
    "a # 数值取决于内存空间的状态，print时候可能overflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 2., 3.],\n",
       "        [4., 5., 6.]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 用list的数据创建tensor\n",
    "b = t.Tensor([[1,2,3],[4,5,6]])\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.tolist() # 把tensor转为list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`tensor.size()`返回`torch.Size`对象，它是tuple的子类，但其使用方式与tuple略有区别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b_size = b.size()\n",
    "b_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.numel() # b中元素总个数，2*3，等价于b.nelement()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[-8.5771e-19,  4.5622e-41, -8.5256e-19],\n",
       "         [ 4.5622e-41, -2.7717e-24,  4.5622e-41]]), tensor([2., 3.]))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 创建一个和b形状一样的tensor\n",
    "c = t.Tensor(b_size)\n",
    "# 创建一个元素为2和3的tensor\n",
    "d = t.Tensor((2, 3))\n",
    "c, d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了`tensor.size()`，还可以利用`tensor.shape`直接查看tensor的形状，`tensor.shape`等价于`tensor.size()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需要注意的是，`t.Tensor(*sizes)`创建tensor时，系统不会马上分配空间，只是会计算剩余的内存是否足够使用，使用到tensor时才会分配，而其它操作都是在创建完tensor之后马上进行空间分配。其它常用的创建tensor的方法举例如下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.ones(2, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.],\n",
       "        [0., 0., 0.]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.zeros(2, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1, 3, 5])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.arange(1, 6, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 1.0000,  5.5000, 10.0000])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.linspace(1, 10, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1163, -0.1111, -0.0152],\n",
       "        [-0.3127,  2.1007,  0.1645]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.randn(2, 3, device=t.device('cpu'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([4, 3, 1, 2, 0])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.randperm(5) # 长度为5的随机排列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1, 0, 0],\n",
       "        [0, 1, 0]], dtype=torch.int32)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.eye(2, 3, dtype=t.int) # 对角线为1, 不要求行列数一致"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`torch.tensor`是在0.4版本新增加的一个新版本的创建tensor方法，使用的方法，和参数几乎和`np.array`完全一致"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scalar: tensor(3.1416), shape of sclar: torch.Size([])\n"
     ]
    }
   ],
   "source": [
    "scalar = t.tensor(3.14159) \n",
    "print('scalar: %s, shape of sclar: %s' %(scalar, scalar.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vector: tensor([1, 2]), shape of vector: torch.Size([2])\n"
     ]
    }
   ],
   "source": [
    "vector = t.tensor([1, 2])\n",
    "print('vector: %s, shape of vector: %s' %(vector, vector.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 2])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor = t.Tensor(1,2) # 注意和t.tensor([1, 2])的区别\n",
    "tensor.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[0.1000, 1.2000],\n",
       "         [2.2000, 3.1000],\n",
       "         [4.9000, 5.2000]]), torch.Size([3, 2]))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix = t.tensor([[0.1, 1.2], [2.2, 3.1], [4.9, 5.2]])\n",
    "matrix,matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.1111, 0.2222, 0.3333]], dtype=torch.float64)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.tensor([[0.11111, 0.222222, 0.3333333]],\n",
    "                     dtype=t.float64,\n",
    "                     device=t.device('cpu'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([0])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "empty_tensor = t.tensor([])\n",
    "empty_tensor.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 常用Tensor操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过`tensor.view`方法可以调整tensor的形状，但必须保证调整前后元素总数一致。`view`不会修改自身的数据，返回的新tensor与源tensor共享内存，也即更改其中的一个，另外一个也会跟着改变。在实际应用中可能经常需要添加或减少某一维度，这时候`squeeze`和`unsqueeze`两个函数就派上用场了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 1, 2],\n",
       "        [3, 4, 5]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 6)\n",
    "a.view(2, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.view(-1, 3) # 当某一维为-1的时候，会自动计算它的大小\n",
    "b.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 1, 3])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.unsqueeze(1) # 注意形状，在第1维（下标从0开始）上增加“１” \n",
    "#等价于 b[:,None]\n",
    "b[:, None].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[0, 1, 2]],\n",
       "\n",
       "        [[3, 4, 5]]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.unsqueeze(-2) # -2表示倒数第二个维度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[0, 1, 2],\n",
       "          [3, 4, 5]]]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = b.view(1, 1, 1, 2, 3)\n",
    "c.squeeze(0) # 压缩第0维的“１”"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 1, 2],\n",
       "        [3, 4, 5]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.squeeze() # 把所有维度为“1”的压缩"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0, 100,   2],\n",
       "        [  3,   4,   5]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[1] = 100\n",
    "b # a修改，b作为view之后的，也会跟着修改"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`resize`是另一种可用来调整`size`的方法，但与`view`不同，它可以修改tensor的大小。如果新大小超过了原大小，会自动分配新的内存空间，而如果新大小小于原大小，则之前的数据依旧会被保存，看一个例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0, 100,   2]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.resize_(1, 3)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0, 100,   2],\n",
       "        [  3,   4,   5],\n",
       "        [  0,   0, 112]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.resize_(3, 3) # 旧的数据依旧保存着，多出的大小会分配新空间\n",
    "b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 索引操作\n",
    "\n",
    "Tensor支持与numpy.ndarray类似的索引操作，语法上也类似，下面通过一些例子，讲解常用的索引操作。如无特殊说明，索引出来的结果与原tensor共享内存，也即修改一个，另一个会跟着修改。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.1741,  1.4335, -0.8156,  0.7622],\n",
       "        [-0.6334, -1.4628, -0.7428,  0.0410],\n",
       "        [-0.6551,  1.0258,  2.0572,  0.3923]])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.randn(3, 4)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 1.1741,  1.4335, -0.8156,  0.7622])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0] # 第0行(下标从0开始)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 1.1741, -0.6334, -0.6551])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:, 0] # 第0列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(-0.8156)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0][2] # 第0行第2个元素，等价于a[0, 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.7622)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0, -1] # 第0行最后一个元素"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.1741,  1.4335, -0.8156,  0.7622],\n",
       "        [-0.6334, -1.4628, -0.7428,  0.0410]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:2] # 前两行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.1741,  1.4335],\n",
       "        [-0.6334, -1.4628]])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:2, 0:2] # 前两行，第0,1列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1.1741, 1.4335]])\n",
      "tensor([1.1741, 1.4335])\n"
     ]
    }
   ],
   "source": [
    "print(a[0:1, :2]) # 第0行，前两列 \n",
    "print(a[0, :2]) # 注意两者的区别：形状不同"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 3, 4])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# None类似于np.newaxis, 为a新增了一个轴\n",
    "# 等价于a.view(1, a.shape[0], a.shape[1])\n",
    "a[None].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 3, 4])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[None].shape # 等价于a[None,:,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 1, 4])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:,None,:].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 1, 4, 1, 1])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:,None,:,None,None].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1, 1, 0, 0],\n",
       "        [0, 0, 0, 0],\n",
       "        [0, 1, 1, 0]], dtype=torch.uint8)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a > 1 # 返回一个ByteTensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1.1741, 1.4335, 1.0258, 2.0572])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[a>1] # 等价于a.masked_select(a>1)\n",
    "# 选择结果与原tensor不共享内存空间"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.1741,  1.4335, -0.8156,  0.7622],\n",
       "        [-0.6334, -1.4628, -0.7428,  0.0410]])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[t.LongTensor([0,1])] # 第0行和第1行"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其它常用的选择函数如表3-2所示。\n",
    "\n",
    "表3-2常用的选择函数\n",
    "\n",
    "函数|功能|\n",
    ":---:|:---:|\n",
    "index_select(input, dim, index)|在指定维度dim上选取，比如选取某些行、某些列\n",
    "masked_select(input, mask)|例子如上，a[a>0]，使用ByteTensor进行选取\n",
    "non_zero(input)|非0元素的下标\n",
    "gather(input, dim, index)|根据index，在dim维度上选取数据，输出的size与index一样\n",
    "\n",
    "\n",
    "`gather`是一个比较复杂的操作，对一个2维tensor，输出的每个元素如下：\n",
    "\n",
    "```python\n",
    "out[i][j] = input[index[i][j]][j]  # dim=0\n",
    "out[i][j] = input[i][index[i][j]]  # dim=1\n",
    "```\n",
    "三维tensor的`gather`操作同理，下面举几个例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0,  1,  2,  3],\n",
       "        [ 4,  5,  6,  7],\n",
       "        [ 8,  9, 10, 11],\n",
       "        [12, 13, 14, 15]])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 16).view(4, 4)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0,  5, 10, 15]])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取对角线的元素\n",
    "index = t.LongTensor([[0,1,2,3]])\n",
    "a.gather(0, index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 3],\n",
       "        [ 6],\n",
       "        [ 9],\n",
       "        [12]])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取反对角线上的元素\n",
    "index = t.LongTensor([[3,2,1,0]]).t()\n",
    "a.gather(1, index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[12,  9,  6,  3]])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取反对角线上的元素，注意与上面的不同\n",
    "index = t.LongTensor([[3,2,1,0]])\n",
    "a.gather(0, index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0,  3],\n",
       "        [ 5,  6],\n",
       "        [10,  9],\n",
       "        [15, 12]])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取两个对角线上的元素\n",
    "index = t.LongTensor([[0,1,2,3],[3,2,1,0]]).t()\n",
    "b = a.gather(1, index)\n",
    "b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "与`gather`相对应的逆操作是`scatter_`，`gather`把数据从input中按index取出，而`scatter_`是把取出的数据再放回去。注意`scatter_`函数是inplace操作。\n",
    "\n",
    "```python\n",
    "out = input.gather(dim, index)\n",
    "-->近似逆操作\n",
    "out = Tensor()\n",
    "out.scatter_(dim, index)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  0.,  0.,  3.],\n",
       "        [ 0.,  5.,  6.,  0.],\n",
       "        [ 0.,  9., 10.,  0.],\n",
       "        [12.,  0.,  0., 15.]])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 把两个对角线元素放回去到指定位置\n",
    "c = t.zeros(4,4)\n",
    "c.scatter_(1, index, b.float())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对tensor的任何索引操作仍是一个tensor，想要获取标准的python对象数值，需要调用`tensor.item()`, 这个方法只对包含一个元素的tensor适用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0,0] #依旧是tensor）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0,0].item() # python float"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 1, 1])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = a[0:1, 0:1, None]\n",
    "print(d.shape)\n",
    "d.item() # 只包含一个元素的tensor即可调用tensor.item,与形状无关"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a[0].item()  ->\n",
    "# raise ValueError: only one element tensors can be converted to Python scalars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 高级索引\n",
    "PyTorch在0.2版本中完善了索引操作，目前已经支持绝大多数numpy的高级索引[^10]。高级索引可以看成是普通索引操作的扩展，但是高级索引操作的结果一般不和原始的Tensor共享内存。 \n",
    "[^10]: https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#advanced-indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0,  1,  2],\n",
       "         [ 3,  4,  5],\n",
       "         [ 6,  7,  8]],\n",
       "\n",
       "        [[ 9, 10, 11],\n",
       "         [12, 13, 14],\n",
       "         [15, 16, 17]],\n",
       "\n",
       "        [[18, 19, 20],\n",
       "         [21, 22, 23],\n",
       "         [24, 25, 26]]])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.arange(0,27).view(3,3,3)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([14, 24])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[[1, 2], [1, 2], [2, 0]] # x[1,1,2]和x[2,2,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([19, 10,  1])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[[2, 1, 0], [0], [1]] # x[2,0,1],x[1,0,1],x[0,0,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0,  1,  2],\n",
       "         [ 3,  4,  5],\n",
       "         [ 6,  7,  8]],\n",
       "\n",
       "        [[18, 19, 20],\n",
       "         [21, 22, 23],\n",
       "         [24, 25, 26]]])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[[0, 2], ...] # x[0] 和 x[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tensor类型\n",
    "\n",
    "Tensor有不同的数据类型，如表3-3所示，每种类型分别对应有CPU和GPU版本(HalfTensor除外)。默认的tensor是FloatTensor，可通过`t.set_default_tensor_type` 来修改默认tensor类型(如果默认类型为GPU tensor，则所有操作都将在GPU上进行)。Tensor的类型对分析内存占用很有帮助。例如对于一个size为(1000, 1000, 1000)的FloatTensor，它有`1000*1000*1000=10^9`个元素，每个元素占32bit/8 = 4Byte内存，所以共占大约4GB内存/显存。HalfTensor是专门为GPU版本设计的，同样的元素个数，显存占用只有FloatTensor的一半，所以可以极大缓解GPU显存不足的问题，但由于HalfTensor所能表示的数值大小和精度有限[^2]，所以可能出现溢出等问题。\n",
    "\n",
    "[^2]: https://stackoverflow.com/questions/872544/what-range-of-numbers-can-be-represented-in-a-16-32-and-64-bit-ieee-754-syste\n",
    "\n",
    "表3-3: tensor数据类型\n",
    "\n",
    "| Data type                | dtype                             | CPU tensor                                                   | GPU tensor                |\n",
    "| ------------------------ | --------------------------------- | ------------------------------------------------------------ | ------------------------- |\n",
    "| 32-bit floating point    | `torch.float32` or `torch.float`  | `torch.FloatTensor`                                          | `torch.cuda.FloatTensor`  |\n",
    "| 64-bit floating point    | `torch.float64` or `torch.double` | `torch.DoubleTensor`                                         | `torch.cuda.DoubleTensor` |\n",
    "| 16-bit floating point    | `torch.float16` or `torch.half`   | `torch.HalfTensor`                                           | `torch.cuda.HalfTensor`   |\n",
    "| 8-bit integer (unsigned) | `torch.uint8`                     | [`torch.ByteTensor`](https://pytorch.org/docs/stable/tensors.html#torch.ByteTensor) | `torch.cuda.ByteTensor`   |\n",
    "| 8-bit integer (signed)   | `torch.int8`                      | `torch.CharTensor`                                           | `torch.cuda.CharTensor`   |\n",
    "| 16-bit integer (signed)  | `torch.int16` or `torch.short`    | `torch.ShortTensor`                                          | `torch.cuda.ShortTensor`  |\n",
    "| 32-bit integer (signed)  | `torch.int32` or `torch.int`      | `torch.IntTensor`                                            | `torch.cuda.IntTensor`    |\n",
    "| 64-bit integer (signed)  | `torch.int64` or `torch.long`     | `torch.LongTensor`                                           | `torch.cuda.LongTensor`   |\n",
    "\n",
    " \n",
    "\n",
    "各数据类型之间可以互相转换，`type(new_type)`是通用的做法，同时还有`float`、`long`、`half`等快捷方法。CPU tensor与GPU tensor之间的互相转换通过`tensor.cuda`和`tensor.cpu`方法实现，此外还可以使用`tensor.to(device)`。Tensor还有一个`new`方法，用法与`t.Tensor`一样，会调用该tensor对应类型的构造函数，生成与当前tensor类型一致的tensor。`torch.*_like(tensora)` 可以生成和`tensora`拥有同样属性(类型，形状，cpu/gpu)的新tensor。 `tensor.new_*(new_shape)` 新建一个不同形状的tensor。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 设置默认tensor，注意参数是字符串\n",
    "t.set_default_tensor_type('torch.DoubleTensor')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.float64"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.Tensor(2,3)\n",
    "a.dtype # 现在a是DoubleTensor,dtype是float64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 恢复之前的默认设置\n",
    "t.set_default_tensor_type('torch.FloatTensor')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.float32"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 把a转成FloatTensor，等价于b=a.type(t.FloatTensor)\n",
    "b = a.float() \n",
    "b.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.],\n",
       "        [0., 0., 0.]])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = a.type_as(b)\n",
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[7.2148e-313, 1.1459e-312, 1.2095e-312],\n",
       "        [1.0610e-312, 1.4217e-312, 1.4005e-312]], dtype=torch.float64)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.new(2,3) # 等价于torch.DoubleTensor(2,3)，建议使用a.new_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.],\n",
       "        [0., 0., 0.]], dtype=torch.float64)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.zeros_like(a) #等价于t.zeros(a.shape,dtype=a.dtype,device=a.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 0, 0],\n",
       "        [0, 0, 0]], dtype=torch.int16)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.zeros_like(a, dtype=t.int16) #可以修改某些属性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.3111, 0.3843, 0.1729],\n",
       "        [0.2693, 0.6378, 0.9917]], dtype=torch.float64)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.rand_like(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1, 1, 1, 1, 1],\n",
       "        [1, 1, 1, 1, 1],\n",
       "        [1, 1, 1, 1, 1],\n",
       "        [1, 1, 1, 1, 1]], dtype=torch.int32)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.new_ones(4,5, dtype=t.int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([3., 4.], dtype=torch.float64)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.new_tensor([3,4]) # "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 逐元素操作\n",
    "\n",
    "这部分操作会对tensor的每一个元素(point-wise，又名element-wise)进行操作，此类操作的输入与输出形状一致。常用的操作如表3-4所示。\n",
    "\n",
    "表3-4: 常见的逐元素操作\n",
    "\n",
    "|函数|功能|\n",
    "|:--:|:--:|\n",
    "|abs/sqrt/div/exp/fmod/log/pow..|绝对值/平方根/除法/指数/求余/求幂..|\n",
    "|cos/sin/asin/atan2/cosh..|相关三角函数|\n",
    "|ceil/round/floor/trunc| 上取整/四舍五入/下取整/只保留整数部分|\n",
    "|clamp(input, min, max)|超过min和max部分截断|\n",
    "|sigmod/tanh..|激活函数\n",
    "\n",
    "对于很多操作，例如div、mul、pow、fmod等，PyTorch都实现了运算符重载，所以可以直接使用运算符。如`a ** 2` 等价于`torch.pow(a,2)`, `a * 2`等价于`torch.mul(a,2)`。\n",
    "\n",
    "其中`clamp(x, min, max)`的输出满足以下公式：\n",
    "$$\n",
    "y_i =\n",
    "\\begin{cases}\n",
    "min,  & \\text{if  } x_i \\lt min \\\\\n",
    "x_i,  & \\text{if  } min \\le x_i \\le max  \\\\\n",
    "max,  & \\text{if  } x_i \\gt max\\\\\n",
    "\\end{cases}\n",
    "$$\n",
    "`clamp`常用在某些需要比较大小的地方，如取一个tensor的每个元素与另一个数的较大值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.0000,  0.5403, -0.4161],\n",
       "        [-0.9900, -0.6536,  0.2837]])"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 6).view(2, 3).float()\n",
    "t.cos(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 1., 2.],\n",
       "        [0., 1., 2.]])"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a % 3 # 等价于t.fmod(a, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  1.,  4.],\n",
       "        [ 9., 16., 25.]])"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a ** 2 # 等价于t.pow(a, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0., 1., 2.],\n",
      "        [3., 4., 5.]])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[3., 3., 3.],\n",
       "        [3., 4., 5.]])"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 取a中的每一个元素与3相比较大的一个 (小于3的截断成3)\n",
    "print(a)\n",
    "t.clamp(a, min=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0000,  0.8415,  0.9093],\n",
       "        [ 0.1411, -0.7568, -0.9589]])"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.sin_() # 效果同 a = a.sin();b=a ,但是更高效节省显存\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  归并操作 \n",
    "此类操作会使输出形状小于输入形状，并可以沿着某一维度进行指定操作。如加法`sum`，既可以计算整个tensor的和，也可以计算tensor中每一行或每一列的和。常用的归并操作如表3-5所示。\n",
    "\n",
    "表3-5: 常用归并操作\n",
    "\n",
    "|函数|功能|\n",
    "|:---:|:---:|\n",
    "|mean/sum/median/mode|均值/和/中位数/众数|\n",
    "|norm/dist|范数/距离|\n",
    "|std/var|标准差/方差|\n",
    "|cumsum/cumprod|累加/累乘|\n",
    "\n",
    "以上大多数函数都有一个参数**`dim`**，用来指定这些操作是在哪个维度上执行的。关于dim(对应于Numpy中的axis)的解释众说纷纭，这里提供一个简单的记忆方式：\n",
    "\n",
    "假设输入的形状是(m, n, k)\n",
    "\n",
    "- 如果指定dim=0，输出的形状就是(1, n, k)或者(n, k)\n",
    "- 如果指定dim=1，输出的形状就是(m, 1, k)或者(m, k)\n",
    "- 如果指定dim=2，输出的形状就是(m, n, 1)或者(m, n)\n",
    "\n",
    "size中是否有\"1\"，取决于参数`keepdim`，`keepdim=True`会保留维度`1`。注意，以上只是经验总结，并非所有函数都符合这种形状变化方式，如`cumsum`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2., 2., 2.]])"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.ones(2, 3)\n",
    "b.sum(dim = 0, keepdim=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2., 2., 2.])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# keepdim=False，不保留维度\"1\"，注意形状\n",
    "b.sum(dim=0, keepdim=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([3., 3.])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.sum(dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5]])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[ 0,  1,  3],\n",
       "        [ 3,  7, 12]])"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 6).view(2, 3)\n",
    "print(a)\n",
    "a.cumsum(dim=1) # 沿着行累加"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 比较\n",
    "比较函数中有一些是逐元素比较，操作类似于逐元素操作，还有一些则类似于归并操作。常用比较函数如表3-6所示。\n",
    "\n",
    "表3-6: 常用比较函数\n",
    "\n",
    "|函数|功能|\n",
    "|:--:|:--:|\n",
    "|gt/lt/ge/le/eq/ne|大于/小于/大于等于/小于等于/等于/不等|\n",
    "|topk|最大的k个数|\n",
    "|sort|排序|\n",
    "|max/min|比较两个tensor最大最小值|\n",
    "\n",
    "表中第一行的比较操作已经实现了运算符重载，因此可以使用`a>=b`、`a>b`、`a!=b`、`a==b`，其返回结果是一个`ByteTensor`，可用来选取元素。max/min这两个操作比较特殊，以max来说，它有以下三种使用情况：\n",
    "- t.max(tensor)：返回tensor中最大的一个数\n",
    "- t.max(tensor,dim)：指定维上最大的数，返回tensor和下标\n",
    "- t.max(tensor1, tensor2): 比较两个tensor相比较大的元素\n",
    "\n",
    "至于比较一个tensor和一个数，可以使用clamp函数。下面举例说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  3.,  6.],\n",
       "        [ 9., 12., 15.]])"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.linspace(0, 15, 6).view(2, 3)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[15., 12.,  9.],\n",
       "        [ 6.,  3.,  0.]])"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.linspace(15, 0, 6).view(2, 3)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 0, 0],\n",
       "        [1, 1, 1]], dtype=torch.uint8)"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a>b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 9., 12., 15.])"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[a>b] # a中大于b的元素"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(15.)"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.max(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.return_types.max(values=tensor([15.,  6.]), indices=tensor([0, 0]))"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.max(b, dim=1) \n",
    "# 第一个返回值的15和6分别表示第0行和第1行最大的元素\n",
    "# 第二个返回值的0和0表示上述最大的数是该行第0个元素"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[15., 12.,  9.],\n",
       "        [ 9., 12., 15.]])"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.max(a,b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[10., 10., 10.],\n",
       "        [10., 12., 15.]])"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 比较a和10较大的元素\n",
    "t.clamp(a, min=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 线性代数\n",
    "\n",
    "PyTorch的线性函数主要封装了Blas和Lapack，其用法和接口都与之类似。常用的线性代数函数如表3-7所示。\n",
    "\n",
    "表3-7: 常用的线性代数函数\n",
    "\n",
    "|函数|功能|\n",
    "|:---:|:---:|\n",
    "|trace|对角线元素之和(矩阵的迹)|\n",
    "|diag|对角线元素|\n",
    "|triu/tril|矩阵的上三角/下三角，可指定偏移量|\n",
    "|mm/bmm|矩阵乘法，batch的矩阵乘法|\n",
    "|addmm/addbmm/addmv/addr/badbmm..|矩阵运算\n",
    "|t|转置|\n",
    "|dot/cross|内积/外积\n",
    "|inverse|求逆矩阵\n",
    "|svd|奇异值分解\n",
    "\n",
    "具体使用说明请参见官方文档[^3]，需要注意的是，矩阵的转置会导致存储空间不连续，需调用它的`.contiguous`方法将其转为连续。\n",
    "[^3]: http://pytorch.org/docs/torch.html#blas-and-lapack-operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.t()\n",
    "b.is_contiguous()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  9.],\n",
       "        [ 3., 12.],\n",
       "        [ 6., 15.]])"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.contiguous()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.2 Tensor和Numpy\n",
    "\n",
    "Tensor和Numpy数组之间具有很高的相似性，彼此之间的互操作也非常简单高效。需要注意的是，Numpy和Tensor共享内存。由于Numpy历史悠久，支持丰富的操作，所以当遇到Tensor不支持的操作时，可先转成Numpy数组，处理后再转回tensor，其转换开销很小。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 1., 1.],\n",
       "       [1., 1., 1.]], dtype=float32)"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "a = np.ones([2, 3],dtype=np.float32)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.from_numpy(a)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.Tensor(a) # 也可以直接将numpy对象传入Tensor\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  1., 100.,   1.],\n",
       "        [  1.,   1.,   1.]])"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0, 1]=100\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1., 100.,   1.],\n",
       "       [  1.,   1.,   1.]], dtype=float32)"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = b.numpy() # a, b, c三个对象共享内存\n",
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意**： 当numpy的数据类型和Tensor的类型不一样的时候，数据会被复制，不会共享内存。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.ones([2, 3])\n",
    "# 注意和上面的a的区别（dtype不是float32）\n",
    "a.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.float32"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.Tensor(a) # 此处进行拷贝，不共享内存\n",
    "b.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]], dtype=torch.float64)"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = t.from_numpy(a) # 注意c的类型（DoubleTensor）\n",
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0, 1] = 100\n",
    "b # b与a不共享内存，所以即使a改变了，b也不变"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  1., 100.,   1.],\n",
       "        [  1.,   1.,   1.]], dtype=torch.float64)"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c # c与a共享内存"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意：** 不论输入的类型是什么，t.tensor都会进行数据拷贝，不会共享内存"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor = t.tensor(a) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1., 100.,   1.],\n",
       "       [  1.,   1.,   1.]])"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor[0,0]=0\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "广播法则(broadcast)是科学运算中经常使用的一个技巧，它在快速执行向量化的同时不会占用额外的内存/显存。\n",
    "Numpy的广播法则定义如下：\n",
    "\n",
    "- 让所有输入数组都向其中shape最长的数组看齐，shape中不足的部分通过在前面加1补齐\n",
    "- 两个数组要么在某一个维度的长度一致，要么其中一个为1，否则不能计算 \n",
    "- 当输入数组的某个维度的长度为1时，计算时沿此维度复制扩充成一样的形状\n",
    "\n",
    "PyTorch当前已经支持了自动广播法则，但是笔者还是建议读者通过以下两个函数的组合手动实现广播法则，这样更直观，更不易出错：\n",
    "\n",
    "- `unsqueeze`或者`view`，或者tensor[None],：为数据某一维的形状补1，实现法则1\n",
    "- `expand`或者`expand_as`，重复数组，实现法则3；该操作不会复制数组，所以不会占用额外的空间。\n",
    "\n",
    "注意，repeat实现与expand相类似的功能，但是repeat会把相同数据复制多份，因此会占用额外的空间。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "a = t.ones(3, 2)\n",
    "b = t.zeros(2, 3,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[1., 1.],\n",
       "         [1., 1.],\n",
       "         [1., 1.]],\n",
       "\n",
       "        [[1., 1.],\n",
       "         [1., 1.],\n",
       "         [1., 1.]]])"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 自动广播法则\n",
    "# 第一步：a是2维,b是3维，所以先在较小的a前面补1 ，\n",
    "#               即：a.unsqueeze(0)，a的形状变成（1，3，2），b的形状是（2，3，1）,\n",
    "# 第二步:   a和b在第一维和第三维形状不一样，其中一个为1 ，\n",
    "#               可以利用广播法则扩展，两个形状都变成了（2，3，2）\n",
    "a+b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[1., 1.],\n",
       "         [1., 1.],\n",
       "         [1., 1.]],\n",
       "\n",
       "        [[1., 1.],\n",
       "         [1., 1.],\n",
       "         [1., 1.]]])"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 手动广播法则\n",
    "# 或者 a.view(1,3,2).expand(2,3,2)+b.expand(2,3,2)\n",
    "a[None].expand(2, 3, 2) + b.expand(2,3,2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "# expand不会占用额外空间，只会在需要的时候才扩充，可极大节省内存\n",
    "e = a.unsqueeze(0).expand(10000000000000, 3,2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.3 内部结构\n",
    "\n",
    "tensor的数据结构如图3-1所示。tensor分为头信息区(Tensor)和存储区(Storage)，信息区主要保存着tensor的形状（size）、步长（stride）、数据类型（type）等信息，而真正的数据则保存成连续数组。由于数据动辄成千上万，因此信息区元素占用内存较少，主要内存占用则取决于tensor中元素的数目，也即存储区的大小。\n",
    "\n",
    "一般来说一个tensor有着与之相对应的storage, storage是在data之上封装的接口，便于使用，而不同tensor的头信息一般不同，但却可能使用相同的数据。下面看两个例子。\n",
    "\n",
    "![图3-1: Tensor的数据结构](imgs/tensor_data_structure.svg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0\n",
       " 1\n",
       " 2\n",
       " 3\n",
       " 4\n",
       " 5\n",
       "[torch.LongStorage of size 6]"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 6)\n",
    "a.storage()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0\n",
       " 1\n",
       " 2\n",
       " 3\n",
       " 4\n",
       " 5\n",
       "[torch.LongStorage of size 6]"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.view(2, 3)\n",
    "b.storage()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 一个对象的id值可以看作它在内存中的地址\n",
    "# storage的内存地址一样，即是同一个storage\n",
    "id(b.storage()) == id(a.storage())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0, 100,   2],\n",
       "        [  3,   4,   5]])"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# a改变，b也随之改变，因为他们共享storage\n",
    "a[1] = 100\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0\n",
       " 100\n",
       " 2\n",
       " 3\n",
       " 4\n",
       " 5\n",
       "[torch.LongStorage of size 6]"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = a[2:] \n",
    "c.storage()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(61277776, 61277760)"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.data_ptr(), a.data_ptr() # data_ptr返回tensor首元素的内存地址\n",
    "# 可以看出相差8，这是因为2*4=8--相差两个元素，每个元素占4个字节(float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([   0,  100, -100,    3,    4,    5])"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c[0] = -100 # c[0]的内存地址对应a[2]的内存地址\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[6666,  100, -100],\n",
       "        [   3,    4,    5]])"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = t.LongTensor(c.storage())\n",
    "d[0] = 6666\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 下面４个tensor共享storage\n",
    "id(a.storage()) == id(b.storage()) == id(c.storage()) == id(d.storage())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 2, 0)"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.storage_offset(), c.storage_offset(), d.storage_offset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e = b[::2, ::2] # 隔2行/列取一个元素\n",
    "id(e.storage()) == id(a.storage())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3, 1), (6, 2))"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.stride(), e.stride()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.is_contiguous()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见绝大多数操作并不修改tensor的数据，而只是修改了tensor的头信息。这种做法更节省内存，同时提升了处理速度。在使用中需要注意。\n",
    "此外有些操作会导致tensor不连续，这时需调用`tensor.contiguous`方法将它们变成连续的数据，该方法会使数据复制一份，不再与原来的数据共享storage。\n",
    "另外读者可以思考一下，之前说过的高级索引一般不共享stroage，而普通索引共享storage，这是为什么？（提示：普通索引可以通过只修改tensor的offset，stride和size，而不修改storage来实现）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.4 其它有关Tensor的话题\n",
    "这部分的内容不好专门划分一小节，但是笔者认为仍值得读者注意，故而将其放在这一小节。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### GPU/CPU\n",
    "tensor可以很随意的在gpu/cpu上传输。使用`tensor.cuda(device_id)`或者`tensor.cpu()`。另外一个更通用的方法是`tensor.to(device)`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.randn(3, 4)\n",
    "a.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "if t.cuda.is_available():\n",
    "    a = t.randn(3,4, device=t.device('cuda:1'))\n",
    "    # 等价于\n",
    "    # a.t.randn(3,4).cuda(1)\n",
    "    # 但是前者更快\n",
    "    a.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.8934,  0.8776,  2.3714, -0.0861],\n",
       "        [-0.2218,  1.7379,  0.5166,  0.2940],\n",
       "        [ 1.1621,  0.6702, -0.4791, -0.7298]])"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = t.device('cpu')\n",
    "a.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意**\n",
    "- 尽量使用`tensor.to(device)`, 将`device`设为一个可配置的参数，这样可以很轻松的使程序同时兼容GPU和CPU\n",
    "- 数据在GPU之中传输的速度要远快于内存(CPU)到显存(GPU), 所以尽量避免频繁的在内存和显存中传输数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 持久化\n",
    "Tensor的保存和加载十分的简单，使用t.save和t.load即可完成相应的功能。在save/load时可指定使用的`pickle`模块，在load时还可将GPU tensor映射到CPU或其它GPU上。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "if t.cuda.is_available():\n",
    "    a = a.cuda(1) # 把a转为GPU1上的tensor,\n",
    "    t.save(a,'a.pth')\n",
    "\n",
    "    # 加载为b, 存储于GPU1上(因为保存时tensor就在GPU1上)\n",
    "    b = t.load('a.pth')\n",
    "    # 加载为c, 存储于CPU\n",
    "    c = t.load('a.pth', map_location=lambda storage, loc: storage)\n",
    "    # 加载为d, 存储于GPU0上\n",
    "    d = t.load('a.pth', map_location={'cuda:1':'cuda:0'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####   向量化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "向量化计算是一种特殊的并行计算方式，相对于一般程序在同一时间只执行一个操作的方式，它可在同一时间执行多个操作，通常是对不同的数据执行同样的一个或一批指令，或者说把指令应用于一个数组/向量上。向量化可极大提高科学运算的效率，Python本身是一门高级语言，使用很方便，但这也意味着很多操作很低效，尤其是`for`循环。在科学计算程序中应当极力避免使用Python原生的`for循环`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "def for_loop_add(x, y):\n",
    "    result = []\n",
    "    for i,j in zip(x, y):\n",
    "        result.append(i + j)\n",
    "    return t.Tensor(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 loops, best of 3: 997 µs per loop\n",
      "The slowest run took 5.80 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
      "10 loops, best of 3: 4.91 µs per loop\n"
     ]
    }
   ],
   "source": [
    "x = t.zeros(100)\n",
    "y = t.ones(100)\n",
    "%timeit -n 10 for_loop_add(x, y)\n",
    "%timeit -n 10 x + y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见二者有超过几十倍的速度差距，因此在实际使用中应尽量调用内建函数(buildin-function)，这些函数底层由C/C++实现，能通过执行底层优化实现高效计算。因此在平时写代码时，就应养成向量化的思维习惯，千万避免对较大的tensor进行逐元素遍历。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外还有以下几点需要注意：\n",
    "- 大多数`t.function`都有一个参数`out`，这时候产生的结果将保存在out指定tensor之中。\n",
    "- `t.set_num_threads`可以设置PyTorch进行CPU多线程并行计算时候所占用的线程数，这个可以用来限制PyTorch所占用的CPU数目。\n",
    "- `t.set_printoptions`可以用来设置打印tensor时的数值精度和格式。\n",
    "下面举例说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(19999999) tensor(19999998)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(tensor(19999999), tensor(19999998))"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 20000000)\n",
    "print(a[-1], a[-2]) # 32bit的IntTensor精度有限导致溢出\n",
    "b = t.LongTensor()\n",
    "t.arange(0, 20000000, out=b) # 64bit的LongTensor不会溢出\n",
    "b[-1],b[-2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.2645,  0.0270, -0.7174],\n",
       "        [ 0.4271,  0.0869,  0.6367]])"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.randn(2,3)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.2645452023,  0.0270472951, -0.7174307108],\n",
       "        [ 0.4270690084,  0.0868919790,  0.6366676688]])"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.set_printoptions(precision=10)\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.5 小试牛刀：线性回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "线性回归是机器学习入门知识，应用十分广泛。线性回归利用数理统计中回归分析，来确定两种或两种以上变量间相互依赖的定量关系的，其表达形式为$y = wx+b+e$，$e$为误差服从均值为0的正态分布。首先让我们来确认线性回归的损失函数：\n",
    "$$\n",
    "loss = \\sum_i^N \\frac 1 2 ({y_i-(wx_i+b)})^2\n",
    "$$\n",
    "然后利用随机梯度下降法更新参数$\\textbf{w}$和$\\textbf{b}$来最小化损失函数，最终学得$\\textbf{w}$和$\\textbf{b}$的数值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch as t\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "from IPython import display\n",
    "\n",
    "device = t.device('cpu') #如果你想用gpu，改成t.device('cuda:0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 设置随机数种子，保证在不同电脑上运行时下面的输出一致\n",
    "t.manual_seed(1000) \n",
    "\n",
    "def get_fake_data(batch_size=8):\n",
    "    ''' 产生随机数据：y=x*2+3，加上了一些噪声'''\n",
    "    x = t.rand(batch_size, 1, device=device) * 5\n",
    "    y = x * 2 + 3 +  t.randn(batch_size, 1, device=device)\n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7f2cb54d0210>"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAEJlJREFUeJzt3W9sXfddx/HPB8elt92YUXOBxm1J\nEcj8aWjdWVHHWDVWirtRtVbogyBt0EkQBIO1PDBaeMA0nkTICPFPYoraocJGt5GlVqnWupXWgXiw\nDLdOl3Spp66sW28GuR24XberNjFfHvjadZxr33Pje+65v3PfL8nq9fWp8zk58ece/87v3J8jQgCA\ndPxA0QEAAJ2huAEgMRQ3ACSG4gaAxFDcAJAYihsAEkNxA0BiKG4ASAzFDQCJ2ZHHN925c2fs3r07\nj28NAKX01FNPvRwR1Szb5lLcu3fv1vz8fB7fGgBKyfaLWbdlqAQAEkNxA0BiKG4ASAzFDQCJobgB\nIDEUNwAkJpfpgAAwKGYXapqZW9TppYZ2jVQ0PTmmqfHRXP9MihsALtLsQk0Hj55Q4+yyJKm21NDB\noyckKdfyZqgEAC7SzNziWmmvapxd1szcYq5/LsUNABfp9FKjo+e7heIGgIu0a6TS0fPdQnEDwEWa\nnhxTZXjovOcqw0OanhzL9c/l4iQAXKTVC5DMKgGAhEyNj+Ze1BsxVAIAiaG4ASAxFDcAJIYxbgB9\nq4jbyVOQ6Yzb9j22T9p+1va9eYcCgNXbyWtLDYXevJ18dqFWdLTCtS1u29dJ+m1JeyVdL+l22z+Z\ndzAAg62o28lTkOWM+2ckHYuI70fEOUn/KmlfvrEADLqibidPQZbiPinpXbavsH2ZpPdJujrfWAAG\nXVG3k6egbXFHxClJfybpcUmPSTouaXnjdrYP2J63PV+v17seFMBgKep28hRkujgZEfdHxNsj4mZJ\n/yvpay22ORwRExExUa1Wu50TwICZGh/VoX17NDpSkSWNjlR0aN8eZpUo43RA2z8SEWdsX6OV8e2b\n8o0FAO1vJx/U6YJZ53F/zvYVks5K+lBELOWYCQDaKmr1mX6Qqbgj4l15BwGATmw1XbDsxc0t7wCS\nNMjTBSluAEka5OmCFDeAJA3ydEHeZApAkopafaYfUNwAklXE6jP9gOIGUDpln99NcQMolUGY383F\nSQClMghvB0txAyiVQZjfTXEDKJVBmN9NcQMolUGY383FSQClMgjzuyluAKVT9vndDJUAQGIobgBI\nDMUNAImhuAEgMRQ3ACSG4gaAxFDcAJAYihsAEpOpuG3/oe1nbZ+0/aDtS/MOBgBorW1x2x6V9GFJ\nExFxnaQhSfvzDgYAaC3rUMkOSRXbOyRdJul0fpEAAFtp+14lEVGz/eeSvimpIenxiHh843a2D0g6\nIEnXXHNNt3MCaKHsS3ShtSxDJT8s6U5J10raJely2+/fuF1EHI6IiYiYqFar3U8K4DyrS3TVlhoK\nvblE1+xCrehoyFmWoZJflvSfEVGPiLOSjkr6hXxjAWhnEJboQmtZivubkm6yfZltS7pF0ql8YwFo\nZxCW6EJrbYs7Io5JOiLpaUknmv/P4ZxzAWhjEJboQmuZZpVExEcj4qcj4rqI+EBEvJ53MABbG4Ql\nutAaK+AAiRqEJbrQGsUNJKzsS3ShNd6rBAASQ3EDQGIYKgF6jLsdsV0UN9BDq3c7rt44s3q3oyTK\nG5lR3CiFVM5it7rbsR/zoj9R3EheSmex3O2IbuDiJJKX0nt2cLcjuoHiRvJSOovlbkd0A0MlSN6u\nkYpqLUq612exWcbZudsR3UBxI3nTk2PnjXFLvT+L7WScnbsdsV0MlSB5U+OjOrRvj0ZHKrKk0ZGK\nDu3b09NyTGmcHenjjBulUPRZbErj7EgfZ9xAFzBbBL1EcQNdwGwR9BJDJUAXMFsEvURxA11S9Dg7\nBgdDJQCQGIobABLTtrhtj9k+vu7jVdv39iIcAOBCbce4I2JR0g2SZHtIUk3SQznnAgBsotOhklsk\nfT0iXswjDACgvU6Le7+kB1t9wfYB2/O25+v1+vaTAQBaylzcti+RdIekf2719Yg4HBETETFRrVa7\nlQ8AsEEnZ9zvlfR0RPx3XmEAAO11Uty/rk2GSQAAvZPpzknbl0u6VdLv5BsHKL9UFjZG/8pU3BHx\nPUlX5JwFKL2UFjZG/+K9SlBq/XZ2u9WCCxQ3sqK4UVr9eHbLggvoBt6rBKXVj8uJseACuoHiRmn1\n49ktCy6gGyhulFY/nt32w8LGSB9j3Cit6cmx88a4pf44u2XBBWwXxY3SYjkxlBXFjVLj7BZlxBg3\nACSG4gaAxFDcAJAYihsAEkNxA0BiKG4ASAzFDQCJobgBIDEUNwAkhuIGgMRQ3ACQGIobABKTqbht\nj9g+Yvs526dsvyPvYACA1rK+O+BfSXosIu6yfYmky3LMBADYQtvitv02STdLuluSIuINSW/kGwsA\nsJksQyXXSqpL+nvbC7bvs315zrkAAJvIUtw7JN0o6e8iYlzS9yR9ZONGtg/Ynrc9X6/XuxwTALAq\nS3G/JOmliDjW/PyIVor8PBFxOCImImKiWq12MyMAYJ22xR0R/yXpW7ZXV1i9RdJXc00FANhU1lkl\nfyDpU80ZJS9I+mB+kQAAW8lU3BFxXNJEzlkAABlw5yQAJIbiBoDEZB3jBjS7UNPM3KJOLzW0a6Si\n6ckxTY2PFh0LGDgUNzKZXajp4NETapxdliTVlho6ePSEJPVVefPigkHAUAkymZlbXCvtVY2zy5qZ\nWywo0YVWX1xqSw2F3nxxmV2oFR0N6CqKG5mcXmp09HwRUnhxAbqB4kYmu0YqHT1fhBReXIBuoLiR\nyfTkmCrDQ+c9Vxke0vTk2Cb/R++l8OICdAPFjUymxkd1aN8ejY5UZEmjIxUd2renry78pfDiAnQD\ns0qQ2dT4aF8V9Uar2ZhVgrKjuHGe1KfT9fuLC9ANFDfWpDJXGxh0jHFjDdPpgDRQ3FjDdDogDRQ3\n1jCdDkgDxY01TKcD0sDFSaxhOh2QBoob52E6HdD/GCoBgMRQ3ACQmExDJba/Iem7kpYlnYsIFg5G\n16R+tybQa52Mcf9SRLycWxIMJO7WBDrHUAkKxd2aQOeyFndIetz2U7YP5BkIg4W7NYHOZS3uX4yI\nGyW9V9KHbN+8cQPbB2zP256v1+tdDYny4m5NoHOZijsias3/npH0kKS9LbY5HBETETFRrVa7mxKl\nxd2aQOfaFrfty22/dfWxpF+RdDLvYBgMKaysA/SbLLNKflTSQ7ZXt/+niHgs11QYKNytCXSmbXFH\nxAuSru9BFgBABkwHBIDEUNwAkBiKGwASQ3EDQGIobgBIDMUNAImhuAEgMRQ3ACSG4gaAxFDcAJAY\nihsAEkNxA0BiKG4ASAzFDQCJobgBIDEUNwAkhuIGgMRQ3ACQGIobABJDcQNAYrKs8i5Jsj0kaV5S\nLSJuzyPM7EJNM3OLOr3U0K6RiqYnx1j9GwA2yFzcku6RdErSD+URZHahpoNHT6hxdlmSVFtq6ODR\nE5JEeQPAOpmGSmxfJelXJd2XV5CZucW10l7VOLusmbnFvP5IAEhS1jHuv5T0R5L+L68gp5caHT0P\nAIOqbXHbvl3SmYh4qs12B2zP256v1+sdB9k1UunoeQAYVFnOuN8p6Q7b35D0aUnvsf3JjRtFxOGI\nmIiIiWq12nGQ6ckxVYaHznuuMjyk6cmxjr8XAJRZ2+KOiIMRcVVE7Ja0X9IXIuL93Q4yNT6qQ/v2\naHSkIksaHano0L49XJgEgA06mVWSu6nxUYoaANroqLgj4ouSvphLEgBAJtw5CQCJobgBIDEUNwAk\nhuIGgMRQ3ACQGIobABJDcQNAYihuAEgMxQ0AiaG4ASAxFDcAJIbiBoDEUNwAkBiKGwASQ3EDQGIo\nbgBIDMUNAImhuAEgMRQ3ACSG4gaAxFDcAJCYtsVt+1LbX7b9jO1nbX+sF8EAAK3tyLDN65LeExGv\n2R6W9O+2H42IL+WcDQDQQtvijoiQ9Frz0+HmR+QZCgCwuUxj3LaHbB+XdEbSExFxrMU2B2zP256v\n1+vdzgkAaMpU3BGxHBE3SLpK0l7b17XY5nBETETERLVa7XZOAEBTR7NKImJJ0pOSbssnDgCgnSyz\nSqq2R5qPK5JulfRc3sEAAK1lmVVypaQHbA9ppeg/GxGP5BsLALCZLLNKviJpvAdZAAAZZDnjHiiz\nCzXNzC3q9FJDu0Yqmp4c09T4aNGxAGANxb3O7EJNB4+eUOPssiSpttTQwaMnJInyBtA3eK+SdWbm\nFtdKe1Xj7LJm5hYLSgQAF6K41zm91OjoeQAoAsW9zq6RSkfPA0ARKO51pifHVBkeOu+5yvCQpifH\nCkoEABfi4uQ6qxcgmVUCoJ9R3BtMjY9S1AD6GkMlAJAYihsAEkNxA0BiKG4ASAzFDQCJ8cqSkl3+\npnZd0otd/8YXb6ekl4sOsU2p70Pq+aX09yH1/FL6+7BV/h+PiEzLh+VS3P3G9nxETBSdYztS34fU\n80vp70Pq+aX096Fb+RkqAYDEUNwAkJhBKe7DRQfogtT3IfX8Uvr7kHp+Kf196Er+gRjjBoAyGZQz\nbgAojVIVt+3bbC/aft72R1p8/W7bddvHmx+/VUTOzdj+hO0ztk9u8nXb/uvm/n3F9o29zriVDPnf\nbfuVdX//f9LrjO3Yvtr2k7a/avtZ2/e02KZvj0PG/H19HGxfavvLtp9p7sPHWmzzg7Y/0zwGx2zv\n7n3S1jLm314XRUQpPiQNSfq6pJ+QdImkZyT97IZt7pb0t0Vn3WIfbpZ0o6STm3z9fZIelWRJN0k6\nVnTmDvO/W9IjRedssw9XSrqx+fitkr7W4t9R3x6HjPn7+jg0/17f0nw8LOmYpJs2bPN7kj7efLxf\n0meKzt1h/m11UZnOuPdKej4iXoiINyR9WtKdBWfqSET8m6T/2WKTOyX9Q6z4kqQR21f2Jl17GfL3\nvYj4dkQ83Xz8XUmnJG18n9++PQ4Z8/e15t/ra81Ph5sfGy/G3SnpgebjI5Juse0eRdxSxvzbUqbi\nHpX0rXWfv6TW/2B/rfnr7RHbV/cmWtdk3cd+9o7mr5CP2v65osNspfnr97hWzpjWS+I4bJFf6vPj\nYHvI9nFJZyQ9ERGbHoOIOCfpFUlX9Dbl5jLkl7bRRWUq7iz+RdLuiPh5SU/ozVds9MbTWrmt93pJ\nfyNptuA8m7L9Fkmfk3RvRLxadJ5Otcnf98chIpYj4gZJV0naa/u6ojN1IkP+bXVRmYq7Jmn9q9ZV\nzefWRMR3IuL15qf3SXp7j7J1S9t97GcR8erqr5AR8XlJw7Z3FhzrAraHtVJ6n4qIoy026evj0C5/\nKsdBkiJiSdKTkm7b8KW1Y2B7h6S3SfpOb9O1t1n+7XZRmYr7PyT9lO1rbV+ilQsWD6/fYMM45B1a\nGf9LycOSfqM5q+EmSa9ExLeLDpWV7R9bHYe0vVcr//766oetme9+Saci4i822axvj0OW/P1+HGxX\nbY80H1ck3SrpuQ2bPSzpN5uP75L0hWhe9Stalvzb7aLSrDkZEeds/76kOa3MMPlERDxr+08lzUfE\nw5I+bPsOSee0chHt7sICt2D7Qa1c8d9p+yVJH9XKhQ1FxMclfV4rMxqel/R9SR8sJmlrGfLfJel3\nbZ+T1JC0v19+2NZ5p6QPSDrRHKOUpD+WdI2UxHHIkr/fj8OVkh6wPaSVF5XPRsQjG36W75f0j7af\n18rP8v7i4l4gS/5tdRF3TgJAYso0VAIAA4HiBoDEUNwAkBiKGwASQ3EDQGIobgBIDMUNAImhuAEg\nMf8Phf8hDVQyykYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 来看看产生的x-y分布\n",
    "x, y = get_fake_data(batch_size=16)\n",
    "plt.scatter(x.squeeze().cpu().numpy(), y.squeeze().cpu().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8lWWe9/HPBamkECDUQAg1ASkG\ng40REZFiGbGMvbvL7s7qOOqiOLv7zDy7zwCCvY2DbcbVHWfGNi0YQEBsgCAqSgqhhCSUhBIC6cm5\nnj8ShmJITk67T875vl8vXiSH+5z7561+c3FVY61FREQ6vy5OFyAiIr6hQBcRCREKdBGREKFAFxEJ\nEQp0EZEQoUAXEQkRCnQRkRChQBcRCREKdBGREBERyJslJyfbtLS0QN5SRMRrLgv7j9ZRfqQOl7X0\njIuib2IMEV1MQO6/cePG/dba3u1dF9BAT0tLY8OGDYG8pYiIx5pclnc2lvD48ny6VNZx6+i+PDwr\ng2G94wNahzGmyJ3rAhroIiKdgbWW1QXlLMzOI3/fETJTk3jupglMTOvpdGltUqCLiJzg29LDzM/O\n5bNtBxjcqxsv3DyBWWP6YUxgule8oUAXEQFKDlXz+LIC3ttUSo9ukfz8itHcfM5goiI6z9wRBbqI\nhLXD1Q28sLqQ1z7biQH+Zcow/mXKMBJjIp0urcMU6CISluoam/ifz4t4dmUhlbUNXJ05kAenj2RA\nUqzTpXlMgS4indL7m0pZnJPP7ooaBiTFMndGOrMzU9p9n8tl+evmPSzOyaP4YA0XjEjmkVmjGD0g\nMQBV+5cCXUQ6nfc3lfLIu5upaWgCoLSihkfe3QzQZqiv3X6A+dm5fFNymFH9E3n9rrFMHtnu9O5O\nQ4EuIp3O4pz8v4f5MTUNTSzOyW810LfuO8LCpXl8mFdG/+4xPP6j8czOTKFrgBYGBYoCXUQ6nd0V\nNW69XlZZy5MrCvj9F8XERUXw0Mx07po0hJjIroEoM+AU6CLS6QxIiqW0lVA/NqBZVdfIr9ds56U1\n22l0ubj9/DTunTqCnnFRgS41oBToItLpzJ2RflIfOkBsZFceuGQkb6wt4qkVW9l/tI7LxvbnoZnp\nDO4V52C1gdNuoBtjXgUuB8qstWNaXlsMXAHUA9uAO621Ff4sVETkmGP95MdmufTvHsOssf15fnUh\n28urmJjWg5duO4vM1B4OVxpYxlrb9gXGTAaOAq+fEOjTgZXW2kZjzKMA1tqH27tZVlaW1eZcIuJL\nm3YdYkF2Hut3HmRo7zjmzczgktF9O8VSfXcZYzZaa7Pau67dFrq1do0xJu2U15ad8O1a4NqOFigi\n4o2iA1Usysnnb9/sITk+iv83ewzXTxxEZNfOs1Tf13zRh34X8HsffI6ISLsOVtXz7MqtvLG2iIgu\nXfjJxSOYM3ko8dEaEvTqCRhj/h1oBN5s45o5wByA1NRUb24nImGstqGJ1z7dyQurC6mqa+T6iYP4\n6bSR9E2Mcbq0oOFxoBtj7qB5sPRi20ZHvLV2CbAEmvvQPb2fiIQnl8vy3qZSHl+Wz+7DtVyc0YeH\nZ2Uwsm+C06UFHY8C3RgzE3gIuNBaW+3bkkREmn28tZz52Xnk7qlkUI9YkuOiWJlXRt7eI27v3RJO\n2h09MMb8DvgcSDfGlBhj7gaeAxKA5caYr4wxL/q5ThEJI7l7Krnt1fXc+sp6jtQ2cOu5gyk/Usf+\nqnosx/dueX9TqdOlBhV3Zrnc2MrLr/ihFhEJc3sO1/BYTgHvbiohMSaS/7hsFLeeN5ipj31EbaPr\npGvb2rslXGlYWEQcV1nbwIurt/HKJzuwFv7xgqH8eMowkro1L9V3d++WcKdAFxHH1De6+N91RTyz\nspCDVfXMPnMAD05PZ1DPbidd197eLdJMgS4iAWetZem3e1n0QR47D1Rz3tBe/OzSUYwd2L3V60+3\nd8vcGemBKrlTUKCLSEB9sfMg87Nz2bSrgpF943ntjolMSe/d5lL9U/du6cgJReFEgS4iAbGt/CiP\nLs1j2ZZ99EmI5tFrxnLtWYPcPmRidmaKArwdCnQR8avyI3U8/WEBv1tfTExEFx68ZCR3XzCEblGK\nH1/TExURv6iub+Tlj3fw64+2Udvo4qazU7lv2giS46OdLi1kKdBFxKeaXJY/bijmieUFlB2pY8YZ\nfXloZgbDesc7XVrIU6CLiE9Ya1mVX8bCpXkU7DvKhNQkXrh5AllpPZ0uLWwo0EXEa5tLDjM/O5fP\ntx8grVc3fnXzBGaO6RdSh0x0Bgp0EfFY8cFqHluWz5++2k3PuCh+ccVobjpnMFER4XvIhJMU6CLS\nYRXV9Ty/qpDfflaEMfCvFw3jny4cRmJMpNOlhTUFuoi4ra6xidc/K+K5VYVU1jZwzYSBPDh9JP27\nawl+MFCgi0i7XC7LX77ZzeKcfEoO1XDhyN7Mm5XBqP6JrV7//qZSrep0gAJdRNr02bb9LMjOY3Pp\nYUb3T+SNu8fxgxHJp73+/U2lJ+27cmzvckCh7mcKdBFpVcG+IyxcmsfKvDIGdI/hievGM/vMFLq0\ns1R/cU7+SZtogfYuDxQFuoicZF9lLU8uL+APG4qJi45g3qwM7jg/jZjIrm69X3uXO0eBLiIAHK1r\nZMlH23jp4x00ulzccf4Q7p06nB5xUR36HO1d7hwFukiYa2hy8dYXxTy9ooD9R+u5fFx/5s5IZ3Cv\nOI8+T3uXO0eBLhKmrLUs27KPR5fmsX1/FWen9eTl20dx5qAkrz5Xe5c7R4EuEoa+3HWIBdm5fLHz\nEMN6x/HSbVlMG9XHZ0v1g3nvcm+nVAbzlEwFukgY2bm/ikU5eWRv3ktyfDS/vGoM12cNIqJreCzV\n93ZKZbBPyVSgi4SBg1X1PPPhVt5YW0Rk1y7cd/EI5kweSlx0eEWAt1Mqg31KZnj92xQJM7UNTbzy\nyQ5eXL2NqvpGrp+Yyv3TRtAnMcbp0hzh7ZTKYJ+SqUAXCUFNLsu7X5bwxPIC9hyuZdqoPsyblcHw\nPgkBryWY+py9nVIZ7FMy2+04M8a8aowpM8Z8e8JrPY0xy40xW1t+7+HfMkXEXR8VlHPZMx8z9+1v\n6JMQzVtzzuXl2yc6FuaPvLuZ0ooaLMf7nN/fVBrwWqB5SmXsKQukOjKl0tv3+5s7IyG/AWae8to8\n4ENr7Qjgw5bvRcRB3+0+zK2vrOP2V9dTVd/Iszdm8t6PJ3Hu0F6O1dRWn7MTZmemsODqsaQkxWKA\nlKRYFlw91u2/MXj7fn9rt8vFWrvGGJN2ystXAlNavv4tsBp42Id1iYibSitqeHxZPu9tKqV7bCT/\neflobjk3legI95bq+1Mw9jl7O6UymKdketqH3tdau6fl671AXx/VIyJuOlzTwK9Wb+PVT3cAMGfy\nUH584XC6dwueQyaCvc851Hg9KGqttcYYe7o/N8bMAeYApKamens7kbBX3+jijbVFPLtyK4eqG7g6\nM4UHpo9kYI9uPr+XtwOa2gYgsDwN9H3GmP7W2j3GmP5A2ekutNYuAZYAZGVlnTb4RaRt1lr+tnkP\niz7IZ9fBaiYN78Ujs0YxJqW7X+7ni0U02gYgsDwN9D8DtwMLW37/k88qEpHvWb/jIL/MzuXr4goy\n+iXwmzsncuHI3j5bqt8aXy2iCeY+51DTbqAbY35H8wBosjGmBPg5zUH+B2PM3UARcJ0/ixQJV4Vl\nR1m4NI8VufvomxjNomvHcc2EgXRt55AJXwjGAU1pmzuzXG48zR9d7ONaRKRF2ZFanl6xlbe+KP57\nn/Ndk4YQGxW4mSsa0Ox8wmNHHpFOoqqukadWFPCDR1fx5rpdNLks8dERpCTFBjTMIfgX0cj3aem/\nSBBobHLxx43NS/XLj9RxYo/K3spar3f082S2igY0Ox8FuoiDrLWszCtj4dI8tpYd5azBPbDWsv9o\n/UnXebOjnzezVUJlQDOY9pPxJ3W5iDjkm5IKbliylrt/u4FGl+XFWybw9j+fx4FTwvwYTwcjg235\nfaAF234y/qQWukiAFR+sZlFOPn/5eje94qL4ryvP4MazU4lsOWTC14OR4T5bJdj3MPclBbpIgFRU\n1/PsykJe/3wnXbsY7rloOP904VASYk5equ/r1ZXhPlslnH6gKdBF/Ky2oYnffraT51cVcqSukR+d\nNZAHLkmnX/fWD5nw9WBkuC+/D6cfaAp0ET9xuSx/+rqUx3IKKK2oYUp6b+bNyiCjX2K77/XlYGSg\nZ6sE2wBkOP1AU6CL+MGnhfuZn53Ld7srOWNAIouuHcek4cmO1ROo2SrBeIhyOE2/VKCL+FDe3koW\nLs1jdX45KUmxPHX9mfxw/AC6+HGpfjC1iIN1ADJUpl+2R4Eu4gN7D9fyxPJ83t5YQnx0BD+7NIPb\nzksjJtK/qzuDrUUcTgOQwUiBLuKFI7UN/Pqj7bz8yXaaXJY7Jw3hnouG0yMuKiD3D7YWcTgNQAYj\nBbqIBxqaXPxu/S6eXrGVA1X1/HD8AObOSGdQT98fMtGWYGsRh9MAZDBSoIt0gLWWnO/28ugH+ezY\nX8U5Q3ry6qWjGD8oyZF6gq1FHE4DkMFIgS7ipo1FB5mfncfGokMM7xPPK7dnMTWjj18PmWhPMLaI\nw2UAMhgp0EXasWN/FY8uzeOD7/bSOyGaBVeP5UdnDSSiq/NbIalFLCdSoIucxoGjdTzz4VbeXLeL\nqIgu3D9tJP9wwRDiooPrfxu1iOWY4PovUyQI1NQ38eqnO/jV6m3UNDRxw8RB3DdtBH0SWl+qLxIs\nFOgiLZpclnc2lvD48nz2VdZxyei+PDwzg+F94p0uTcQtCnQJSoFc/Wit5aOCchYuzSNv7xHGD0ri\nmRsyOWdoL7/cT8RfFOgSdAK5+vHb0sMsWJrLp4UHSO3ZjeduyuSysf0dnbki4ikFugSdQKx+LDlU\nzePLCnhvUylJ3SL5P5eP5pZzBxMV4fzMFRFPKdAl6Phz9ePhmgZeWFXIa5/tBOCfLxzGv0wZRvfY\nyLbfKNIJKNAl6Phj9WNdYxP/83kRz60q5HBNA1dlpvDg9HRStMeIhBCvAt0Ycz/wD4AFNgN3Wmtr\nfVGYhC9frn50uSx/3byHxTl5FB+s4YIRycyblcEZA7r7smSRoOBxoBtjUoCfAKOttTXGmD8ANwC/\n8VFtEkaOzWoprajB0NxCOCYpNpJf/PCMDvefr91+gAXZuXxdcpiMfgm8ftfZTB7Z26v6tBpTgpm3\nXS4RQKwxpgHoBuz2viQJN6fOarGn/HlVfWOHPm/rviM8+kEeK3LL6N89hsd+NJ6rMlPo6uEhE8G2\n57jI6Xgc6NbaUmPMY8AuoAZYZq1d5rPKJGy0NqvlRA1N1q0ZLmWVtTy5Yiu//2IXcVERPDQznbsm\nDfH6kIlg23Nc5HS86XLpAVwJDAEqgD8aY26x1r5xynVzgDkAqampXpQqwcybLgl3Zq+0dU1VXSNL\n1mznpY+3U9/o4rbz0rh36nB6xUe7Xb8n99YpPBJsvOlymQbssNaWAxhj3gXOB04KdGvtEmAJQFZW\n1ql/m5YQ4G2XxOlmtZx6zakam1z8fkMxTy7fyv6jdVw2tj9zZ6STlhznwT9Fx+vz157j6q8XT3mz\nimIXcK4xpptpXlZ3MZDrm7KkM2mrS8Idc2ekE9tGt0hkV3PSDBdrLcu37GPGU2v49/e+Ja1XN979\n8fk8f/MEn4f56erz157jx344llbUYDn+w/H9TaU+v5eEHm/60NcZY94GvgQagU20tMQlvHjbJXHi\nnt6nznLp0S2Sn19xfIbLV8UVzM/OZf2OgwxNjuPXt57F9NF9/bpUP5B7jqu/Xrzh1SwXa+3PgZ/7\nqBbppHzRJdHent5FB6pYnJPPX7/ZQ3J8FP89eww3TBxEZIAOmQjUnuPqrxdvaKWoeM2fx6Adqqrn\nmZVbeWNtERFduvCTqcOZc+Ew4oPskInWeNIXHmxnhErnEvz/V0jQ80eXRG1DE699upMXVhdSVdfI\ndVmDuP+SkfRN7ByHTHg6UByMZ4RK56FAF5/wVZeEy2V5b1Mpjy/LZ/fhWqZm9GHerAxG9k3wQZWB\n42lfuM4IFW8o0CVofLJ1P/Ozc9myp5IxKYk8dt14zh+W7HRZHvGmL1xnhIqnFOjiuNw9lSxYmsea\ngnJSkmJ5+oYzuWLcALp4uFQ/GKgvXJygQBeveboQZs/hGh5fVsA7X5aQEB3Bv186itvOH0x0hHdL\n9YOB+sLFCcbawC3ezMrKshs2bAjY/cT/Th38A4jsYoiPiaCiuqHVgK+sbeDF1dt45ZMdWAu3nz+Y\nf71oOEndopz4R/AbrfgUXzHGbLTWZrV3nVro4pXWBv8aXJZD1Q3AybM7Lh3bn/9dV8QzKws5WFXP\nlWcO4N+mpzOoZ7eA1x0I6guXQFOgi1fcGeSraWjiv/6yhadWFLDzQDXnDe3Fzy4dxdiBOmRCxJcU\n6OIVdzbWAjhYXU9yQhSv3TGRKem9/bpUXyRc6Yhz8Up7G2sdkxQbSfZPLuCijD4KcxE/UQtdvHLq\nQpiEmAiO1jXiOmGsPSaiC7/44RlEnLLvigYNRXxLLXTx2uzMFJY/MJkHLhlJU0uSx0U1t9pTkmJZ\neM247wW1tokV8T210MUrTS7L2xuLeWJ5Afsq65g+ui8Pz8pgWO/4Nt+nbWJFfE+BLh6x1rI6v5wF\nS3Mp2HeUzNQknrtpAhPTerr1fm0TK+J7CnTpsM0lh5mfncvn2w8wuFc3Xrh5ArPG9OvQYKeWxov4\nngJd3FZ8sJrHluXzp6920zMuil9cMZqbzhlMVETHh2K0NF7E9xTo0q7D1Q08v7qQ33y6E2Pgx1OG\n8c9ThpEYE+nxZ2qbWBHfU6DLadU1NvH6Z0U8t6qQytoGrpkwkAenj6R/d990i3izNF5THkW+T4Eu\n3+NyWf7yzW4W5+RTcqiGySN7M29mBqMHJDpdGuD5aUAioU6BLif5fNsBFizN5ZuSw4zun8j/3D2W\nC0b0drqsk2jKo0jrFOgCQMG+IyxcmsfKvDIGdI/hievGM/vMlKA8ZEJTHkVap0APc2WVtTyxvIA/\nbCgmLjqCebMyuOP8NGLc2J/FKZryKNI6BXqYOlrXyJI123lpzXYaXS5uPz+Ne6eOoGdc8B8yoSmP\nIq1ToIeZhiYXb31RzNMrCth/tJ7LxvXnoRnpDO4Vd9r3BNuMEk15FGmdV4FujEkCXgbGABa4y1r7\nuS8KE9+y1rJsyz4e/SCP7eVVnJ3Wk5duyyAztUeb7wvWGSU6DUjk+7xtoT8NfGCtvdYYEwWE5lli\nndyXuw6xIDuXL3YeYljvOF66LYtpo9zbl1wzSkQ6D48D3RjTHZgM3AFgra0H6n1TlvhC0YEqFn2Q\nz9827yE5PppfXjWG67MGfW9f8rZoRolI5+FNC30IUA68ZowZD2wE7rPWVp14kTFmDjAHIDU11Yvb\nibsOVtXzzIdbeXNdERFdunDfxSP4x8lDiY/u+L9uzSgR6Ty8CfQIYAJwr7V2nTHmaWAe8J8nXmSt\nXQIsAcjKyrLf+xTxmdqGJl79dAe/WrWNqvpGrp+Yyv3TRtAnMcbjz2xtRokBLsrwbrFRsA20ioQC\nbwK9BCix1q5r+f5tmgNdAqzJZXlvUymPL8tnz+Fapo3qw8MzMxjRN8Hrz56dmcKGooO8uXYXx34a\nW+CdjaVkDe7pUQgH60CrSGfncaBba/caY4qNMenW2nzgYmCL70oTd6wpKGfB0jxy91QybmB3nrju\nTM4b1qvVaz1tFa/KK+fUv1p5MzCqgVYR//B2lsu9wJstM1y2A3d6X5K4Y8vuShYszeXjrfsZ1DOW\nZ27M5PKx/U+7VN+bVrGvB0Y10CriH14FurX2KyDLR7WIG3ZX1PD4sgLe3VRCYkwk/3HZKG49bzDR\nEW0v1femVezrgVENtIr4R8ePmhFHVNY28OgHeVz02Gr+8s1u5lwwlDVzL+IfLhjabpiDd63iuTPS\niT1lbxdvltr7+vNEpJmW/ge5+kYXb64r4pkPt3KouoGrMlN4cPpIBvbo2Boub1rFvl5qr6X7Iv5h\nrA3cTMKsrCy7YcOGgN2vM7PWkr15L4ty8ig6UM2k4b14ZNYoxqR09+jzTu1Dh+ZW8YKrxypIRYKc\nMWajtbbd7m210IPQFzsP8su/5fJVcQXpfRP4zZ0TuXBkb7eW6p+OWsUioU+BHkS2lR9l4dI8lm/Z\nR9/EaBZdM45rzhpIVx8dMqENrURCmwI9CJQfqePpDwv43frivw8O3jVpCLFRwXvIhIgEHwW6g6rr\nG3lpzQ6WrNlGXaOLm89J5ScXjyA5Ptrp0kSkE1KgO6CxycXbG0t4YnkBZUfqmDWmH3NnpDO0d7zT\npYlIJ6ZAd5MvNpOy1rIqv4wF2XlsLTvKhNQkfnXLBM4a3NNPVYtIOFGgu8EXm0l9U1LB/Oxc1m4/\nyJDkOH518wRmjunn1cwVEZETKdDd4M2y+eKD1SzOyefPX++mV1wU/3XlGdx4diqRHThkQkTEHQp0\nN3iybL6iup7nVhby+udFdOkC91w0nH+6cCgJMZH+KlNEwpwC3Q0dWTZf29DE65/v5LmVhRypa+RH\nZw3k/ktG0r+7Np4SEf9SoLuhtVN7Tt1MyuWy/Pnr3SzOyae0ooYp6b2ZNyuDjH6JTpQsImFIge6G\n9pbNf1a4n/lLc/m2tJIzBiSy6NpxTBqe7GTJIhKGFOhuam3ZfP7eIyxcmsuq/HJSkmJ58vrxXDk+\n5bSHTIiI+JMC3QN7D9fy5PIC/rixmLjoCB6ZlcHt56cRE6ml+iLiHAV6Bxyta+TXH23jpY+30+Sy\n3DlpCPdcNJwecVFOlyYiokB3R0OTi7fW7+KpFVs5UFXPFeMHMHd6Oqm9OnbIhIiIPynQ22CtJee7\nfSz6II/t+6s4Z0hPXr10FOMHJTldmojI9yjQT2Nj0SEWZOeyoegQw/vE8/JtWVw8qo+W6otI0FKg\nn2LH/ioW5+SRvXkvvROimX/VWK7LGkiEluqLSJBToLc4cLSOZ1cW8sbaIqIiuvDTaSP4xwuGEhet\nRyQinUPYp1VtQxOvfLKDF1dvo7qhiesnDuKn00bQJyHG6dJERDrE60A3xnQFNgCl1trLvS8pMJpc\nlne/bD5kYs/hWqaN6su8WekM75PgdGkiIh7xRQv9PiAX6DSblnxUUM6C7Fzy9h5h/KAknrr+TM4Z\n2svpskREvOJVoBtjBgKXAb8EHvBJRX703e7DLFyax8db95PasxvP3ZTJZWP7a+aKiIQEb1voTwEP\nAUHdT1FaUcPjOfm891Up3WMj+T+Xj+bmc1OJjtBSfREJHR4HujHmcqDMWrvRGDOljevmAHMAUlNT\nAd+cz+mOwzUNvLC6kNc+3QnAnMlD+fGU4XSP1SETIhJ6vGmhTwJ+aIy5FIgBEo0xb1hrbznxImvt\nEmAJQFZWlvXF+ZztqW908cbaIp5duZWKmgauykzhwenppLRyIIWISKgw1lrvP6S5hf5v7c1yycrK\nstHXLmr19J+UpFg+nTfVqzqstfz1mz0szsln18FqfjA8mXmzMhiT0t2rzxURcZIxZqO1Nqu96wI+\nD92T8zndsX7HQX6ZncvXxRVk9Evgt3edzeQRyRrwFJGw4ZNAt9auBla7c21Hzud0R2HZURYuzWNF\n7j76Jcaw+NpxXD1hIF11yISIhJmAt9DdOZ/THWVHanlqxVZ+/0Xx399/16QhxEZp5oqIhKeAB3p7\n53O2p6qukZc+3s4Lq7dR3+gCID46gpSkWIW5iIQ1R/Zyae18zvY0Nrn4w4YSnlxRQPmROk7sUdlb\nWevzmTIiIp1N0O8Ja61lxZZ9zHz6Y3723mZSe3YjOT4K1ymTc2oamlick+9MkSIiQSCod1v8uriC\n+dm5rNtxkKHJcbx4y1nMOKMvQx/JbvV6b2fKiIh0ZkEZ6LsOVLN4WT5/+Xo3veKi+O8rz+CGs1OJ\nbDlkwtczZUREQkFQBXpFdT3Prizk9c930rWL4d6pw5kzeSgJMScv1ffVTBkRkVASFIFe29DEbz/b\nyfOrCjla18iPzhrE/ZeMpF/31g+Z8HamjIhIKHI00F0uy5++LuWxnAJKK2q4KL0382aNIr1f+5s3\nejJTRkQklDkW6J8W7md+di7f7a5kTEoii68dx/nDk50qR0Sk0wt4oOftrWTh0jxW55eTkhTL0zec\nyRXjBtBFS/VFRLwS0EAvOVTDpU9/THx0BD+7NIPbzksjJlKrO0VEfCGggV5RXc89k4Zwz9ThJHWL\nCuStRURCXkADfWTfBP7j8tGBvKWISNgI6NL/qIig32lARKTTUsKKiIQIBbqISIhQoIuIhAgFuohI\niFCgi4iECAW6iEiIUKCLiIQIBbqISIgIiv3QA+X9TaXaQ11EQlbYBPr7m0pPOuWotKKGR97dDKBQ\nF5GQ4HGXizFmkDFmlTFmizHmO2PMfb4szNcW5+SfdGQdQE1DE4tz8h2qSETEt7xpoTcCD1prvzTG\nJAAbjTHLrbVbfFSbT+1u5VDptl4XEelsPG6hW2v3WGu/bPn6CJALBG3fxYCk2A69LiLS2fhklosx\nJg3IBNa18mdzjDEbjDEbysvLfXE7j8ydkU7sKYdpxEZ2Ze6MdIcqEhHxLa8D3RgTD7wD/NRaW3nq\nn1trl1hrs6y1Wb179/b2dh6bnZnCgqvHkpIUiwFSkmJZcPVYDYiKSMjwapaLMSaS5jB/01r7rm9K\n8p/ZmSkKcBEJWd7McjHAK0CutfYJ35UkIiKe8KbLZRJwKzDVGPNVy69LfVSXiIh0kMddLtbaTwDj\nw1pERMQL2stFRCREKNBFREKEAl1EJEQo0EVEQoQCXUQkRCjQRURChAJdRCREKNBFREKEAl1EJEQo\n0EVEQoQCXUQkRCjQRURChAJdRCREKNBFREKEAl1EJEQo0EVEQoQCXUQkRCjQRURChAJdRCREKNBF\nREKEAl1EJEQo0EVEQoQCXUQkRCjQRURChFeBboyZaYzJN8YUGmPm+aooERHpOI8D3RjTFXgemAWM\nBm40xoz2VWEiItIx3rTQzwYKrbXbrbX1wFvAlb4pS0REOsqbQE8Bik/4vqTlNRERcUCEv29gjJkD\nzGn5ts4Y862/79lJJAP7nS5UiU+EAAADBklEQVQiSOhZHKdncZyexXHp7lzkTaCXAoNO+H5gy2sn\nsdYuAZYAGGM2WGuzvLhnyNCzOE7P4jg9i+P0LI4zxmxw5zpvuly+AEYYY4YYY6KAG4A/e/F5IiLi\nBY9b6NbaRmPMPUAO0BV41Vr7nc8qExGRDvGqD91amw1kd+AtS7y5X4jRszhOz+I4PYvj9CyOc+tZ\nGGutvwsREZEA0NJ/EZEQEZBA1xYBxxljXjXGlIX79E1jzCBjzCpjzBZjzHfGmPucrskpxpgYY8x6\nY8zXLc/i/zpdk9OMMV2NMZuMMX91uhYnGWN2GmM2G2O+cmemi9+7XFq2CCgALqF58dEXwI3W2i1+\nvXGQMsZMBo4Cr1trxzhdj1OMMf2B/tbaL40xCcBGYHY4/ndhjDFAnLX2qDEmEvgEuM9au9bh0hxj\njHkAyAISrbWXO12PU4wxO4Esa61b8/ED0ULXFgEnsNauAQ46XYfTrLV7rLVftnx9BMglTFca22ZH\nW76NbPkVtoNbxpiBwGXAy07X0tkEItC1RYC0yRiTBmQC65ytxDktXQxfAWXAcmtt2D4L4CngIcDl\ndCFBwALLjDEbW1bdt0mDouIoY0w88A7wU2ttpdP1OMVa22StPZPmFddnG2PCsjvOGHM5UGat3eh0\nLUHiB9baCTTvavuvLV22pxWIQHdriwAJPy39xe8Ab1pr33W6nmBgra0AVgEzna7FIZOAH7b0Hb8F\nTDXGvOFsSc6x1pa2/F4GvEdzF/ZpBSLQtUWAfE/LQOArQK619gmn63GSMaa3MSap5etYmicQ5Dlb\nlTOstY9Yawdaa9NozoqV1tpbHC7LEcaYuJYJAxhj4oDpQJuz4/we6NbaRuDYFgG5wB/CeYsAY8zv\ngM+BdGNMiTHmbqdrcsgk4FaaW2Bftfy61OmiHNIfWGWM+YbmBtBya21YT9cTAPoCnxhjvgbWA3+z\n1n7Q1hu0UlREJERoUFREJEQo0EVEQoQCXUQkRCjQRURChAJdRCREKNBFREKEAl1EJEQo0EVEQsT/\nB58uRTRovRJFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w:  1.91150105 b:  3.04418444633\n"
     ]
    }
   ],
   "source": [
    "# 随机初始化参数\n",
    "w = t.rand(1, 1).to(device)\n",
    "b = t.zeros(1, 1).to(device)\n",
    "\n",
    "lr =0.02 # 学习率\n",
    "\n",
    "for ii in range(500):\n",
    "    x, y = get_fake_data(batch_size=4)\n",
    "    \n",
    "    # forward：计算loss\n",
    "    y_pred = x.mm(w) + b.expand_as(y) # x@W等价于x.mm(w);for python3 only\n",
    "    loss = 0.5 * (y_pred - y) ** 2 # 均方误差\n",
    "    loss = loss.mean()\n",
    "    \n",
    "    # backward：手动计算梯度\n",
    "    dloss = 1\n",
    "    dy_pred = dloss * (y_pred - y)\n",
    "    \n",
    "    dw = x.t().mm(dy_pred)\n",
    "    db = dy_pred.sum()\n",
    "    \n",
    "    # 更新参数\n",
    "    w.sub_(lr * dw)\n",
    "    b.sub_(lr * db)\n",
    "    \n",
    "    if ii%50 ==0:\n",
    "       \n",
    "        # 画图\n",
    "        display.clear_output(wait=True)\n",
    "        x = t.arange(0, 6).view(-1, 1)\n",
    "        y = x.float().mm(w) + b.expand_as(x)\n",
    "        plt.plot(x.cpu().numpy(), y.cpu().numpy()) # predicted\n",
    "        \n",
    "        x2, y2 = get_fake_data(batch_size=32) \n",
    "        plt.scatter(x2.numpy(), y2.numpy()) # true data\n",
    "        \n",
    "        plt.xlim(0, 5)\n",
    "        plt.ylim(0, 13)\n",
    "        plt.show()\n",
    "        plt.pause(0.5)\n",
    "        \n",
    "print('w: ', w.item(), 'b: ', b.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见程序已经基本学出w=2、b=3，并且图中直线和数据已经实现较好的拟合。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "虽然上面提到了许多操作，但是只要掌握了这个例子基本上就可以了，其他的知识，读者日后遇到的时候，可以再看看这部份的内容或者查找对应文档。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
